Integrity Check
The function of the Data Integrity Check is to perform a health check for the index file and backed up data by %edition_name% to ensure the data integrity and restorability. After each data integrity check, the storage statistics will be refreshed.
From version 7.15.6.0 onwards, %edition_name% support Rebuild index feature to enhance the DIC function.
When index files detected below errors:
- Missing (rename or removed)
- Empty (remove all index content to zero size)
- Corrupted (copy the with .xml to .b2b file)
- Containing corrupted entries
- Valid index files from previous backup job or snapshot will be used instead.
- If all index files in all snapshot are missing, the backup set will be treated as a new one and continue with new empty index.
- If all index files are found to be empty, the remaining procedure will be continued with current index.
- If index file is found containing corrupted entries, Rebuild index will rebuild an index from scratch.
!
Please kindly note that:
- Data Integrity Check feature is used to identify and remove any corrupted file(s) on the backup destination(s), so can be backed up again to ensure the restorability. The Data Integrity Check CANNOT fix or repair files that are already corrupted.
- A data integrity check can only be started when there is no manual, scheduled or continuous backup job running (of the corresponding backup set), and vice versa. It is highly recommended to temporarily disable the backup schedule to ensure that no scheduled backup is started while the data integrity check is running.
- After each data integrity check, the storage statistics will be refreshed.
There are 4 options available:
- With the "Run Cyclic Redundancy Check (CRC) during data integrity check" Unselected and "Rebuild index" Unselected option: for checking the index only.
- With the "Run Cyclic Redundancy Check (CRC) during data integrity check" Selected and "Rebuild index" Unselected option: for checking index and integrity of the files against the checksum file generated at the time of the backup job.
- With the "Run Cyclic Redundancy Check (CRC) during data integrity check" Unselected and "Rebuild index" Selected option: for checking the index and rebuild the index.
- With the "Run Cyclic Redundancy Check (CRC) during data integrity check" Selected and "Rebuild index" Selected option: for checking index and integrity of the files against the checksum file generated at the time of the backup job and rebuild index.
When you start the data integrity check, %edition_name% will do the following operation:
With the "Run Cyclic Redundancy Check (CRC) during data integrity check" Unselected and "Rebuild index" Unselected
%edition_name% will perform a comparison of the files/folders on the backup destination(s) with the list of files/folders recorded in the current index file. If the data integrity check finds:
- There are files/folders in the backup destination(s) which do not exist in the current index file, then the extra files/folders will be deleted from the backup destination(s).
For example: If the backup process is interrupted before completion, there will be some incomplete uploaded files or partially uploaded files.
- There are files/folders listed in the current index file which do not exist in the backup destination(s), then the entries for the extra files/folders will be deleted from the current index file.
- The current index file is corrupted, then the corrupted index file will be deleted and %edition_name% will replace it with the index files from the previous backup job or snapshot, therefore the file(s)/folder(s) backed up in the current backup job or current snapshot will be deleted from the backup destination and no longer be recoverable. %edition_name% will try to upload these files again in the subsequent backup job if they still exist on the clients machine.
With the "Run Cyclic Redundancy Check (CRC) during data integrity check" Selected and and "Rebuild index" Unselected
!
Please kindly note that:
- If the CRC (Cycle Redundancy Check) option is enabled, backup data will be streamed from the backup destination (e.g. the cloud storage location or FTP location for example), to the client computer during the CRC check. For user with metered Internet connection, pay close attention to the data charge fee during a data integrity check if CRC is enabled.
- The time required to complete a data integrity check depends on a number of factors, such as the number of files / folders in the backup set(s), bandwidth available on the client computer, hardware specifications of the client computer such as the disk I/O and CPU performance, and if there are other resource intensive job running. So during a data integrity check with CRC enabled, pay attention to the resource usage on the client computer.
%edition_name% will perform check on the integrity of the files on the backup destination(s) against the checksum file generated at the time of the backup job. If there is a discrepancy, this indicates the file(s) on backup destination(s) are corrupted, and then %edition_name% will remove these files from the backup destination(s). If these file(s) still exist on the client machine on the next backup job, %edition_name% will upload the latest copy.
With the "Run Cyclic Redundancy Check (CRC) during data integrity check" Unselected and "Rebuild index" Selected
%edition_name% will perform a comparison of the files/folders on the backup destination(s) with the list of files/folders recorded in the current index file. If the data integrity check finds:
- There are files/folders in the backup destination(s) which do not exist in the current index file, then the extra files/folders will be deleted from the backup destination(s).
For example: If the backup process is interrupted before completion, there will be some incomplete uploaded files or partially uploaded files.
- There are files/folders listed in the current index file which do not exist in the backup destination(s), then the entries for the extra files/folders will be deleted from the current index file.
- The current index file is corrupted, then the corrupted index file will be deleted and %edition_name% will replace it with the index files from the previous backup job or snapshot, therefore the file(s)/folder(s) backed up in the current backup job or current snapshot will be deleted from the backup destination and no longer be recoverable. %edition_name% will try to upload these files again in the subsequent backup job if they still exist on the clients machine.
- The current index file is incorrect with error returned by a data integrity check job, such as "Cannot parse file", or error returned by a backup, such as "Error initializing bptree", etc, then Rebuild index will rebuild an index from scratch by copying entries from an old index to the new index. If corrupted entries are found during the rebuild, they it will be ignored.
With the "Run Cyclic Redundancy Check (CRC) during data integrity check" Selected and "Rebuild index" Selected
!
Please kindly note that:
- If the CRC (Cycle Redundancy Check) option is enabled, backup data will be streamed from the backup destination (e.g. the cloud storage location or FTP location for example), to the client computer during the CRC check. For user with metered Internet connection, pay close attention to the data charge fee during a data integrity check if CRC is enabled.
- The time required to complete a data integrity check depends on a number of factors, such as the number of files / folders in the backup set(s), bandwidth available on the client computer, hardware specifications of the client computer such as the disk I/O and CPU performance, and if there are other resource intensive job running. So during a data integrity check with CRC enabled, pay attention to the resource usage on the client computer.
%edition_name% will perform check on the integrity of the files on the backup destination(s) against the checksum file generated at the time of the backup job. If there is a discrepancy, which indicates the file(s) on backup destination(s) are corrupted, %edition_name% will remove these files from the backup destination(s). If these file(s) still exist on the client machine on the next backup job, %edition_name% will upload the latest copy.
The current index file is incorrect with error returned by a data integrity check job, such as "Cannot parse file", or error returned by a backup, such as "Error initializing bptree", etc, then Rebuild index will rebuild an index from scratch by copying entries from an old index to the new index. If corrupted entries are found during the rebuild, they it will be ignored.
!
It is highly recommended to select both CRC and Rebuild index feature when running a DIC job, if there were errors returned by a previous backup or data integrity check.
To perform the data integrity check:
- Select a backup set from the drop down list. You can choose a specific backup set or All (the default selection).
!
Please kindly note that:
- If you select "All", all backup sets and all destinations will be checked but this will take longer to complete depending on the number of backup sets and destinations.
- If you select a particular backup set, you can select a particular destination or "All" backup destination to check.
- Click on the "Run Cyclic Redundancy Check (CRC) during data integrity check" option if required to verify the integrity of the data in the backup destination. This will require more time to complete.
- Click on the "Rebuild index" option if required to rebuild the index from scratch. This feature will copy entries from old index to the new index. If any entries are corrupted, then they will be ignored.
- Click [Start] to begin.
- In case you need to stop the progress, press the [Stop] button to quit.
- When the data integrity check is completed, the following TEST MODE page (preview mode) will be shown:
If the "Statistics" status indicates "Correct", it means "Data Integrity Check is completed successfully" and there are no corrupted index or data found in the backup set(s), you can click on the [View log] button to check on the logs for details, or click on [Close] to quit.
If the "Statistics" status indicates "Incorrect", it means there is discrepancy between backed up files and the checksum file generated at the time of the backup job and or a issue with the index files. You can also check the value of "Items found in index" and "Data corrupted items" to see details of the discrepancy between backed up files and the index file.
- For the situation that there are files/folders in the backup destination(s) which do not exist in the current index file, such as some incomplete uploaded files or partially uploaded files remaining in the backup destination(s) caused by interruption of the backup process before completion:
- You can click on [Yes] to delete the corrupted data items, checksum incorrect items and index broken data blocks. If the deleted file(s) still exist on the client machine on the next backup job, %edition_name% will upload the latest copy.
- You can click on [No] so no action will be taken, and the corrupted items, backup or restore problems will NOT be fixed.
- You can click on the [View log] button to check the according log, such as "1519821703318, 2018-02-28-12-41-43,INFO,"Removing file \"1518797253825/blocks/2018-02-23-09-05-00/0/000982.bak\" from destination because it does not exist in index"" .
- For the situation that there are files/folders listed in the current index file which do not exist in the backup destination(s):
- You can click on [Yes] to delete the additional index entries. If the deleted file(s) still exist on the client machine on the next backup job, %edition_name% will upload the latest copy.
- You can click on [No] so no action will be taken, and the corrupted items, backup or restore problems will NOT be fixed.
- You can click on the [View log] button to check the according log, such as "1520578431424,2018-03-09-14-53-51,INFO,Utilities,"Removing backup file \"C:\\Users\\Administrator\\Desktop\\in case backup of ahsaycbs\\build\\engine-framework\\custom-obm\\app\\common\\bin\\cbCoreRes_sl.properties\" from index because it does not exist in destination"".
- For the situation that the current index file is corrupted:
- You can click on [Yes] to delete the corrupted files. %edition_name% will replace it with the index files from the previous backup job or snapshot, therefore the file(s)/folder(s) backed up in the current backup job or current snapshot will be deleted from the backup destination and no longer be recoverable. If the deleted file(s) still exist on the client machine on the next backup job, %edition_name% will upload the latest copy.
- You can click on [No] so no action will be taken, and the backup and restore problem will NOT be fixed.
- You can click on the [View log] button to check the according log, such as "1520837556138,2018-03-12-14-52-36,INFO,Utilities,"Index files are corrupted. Download valid index files from backup job \"2018-03-09-18-30-35\"" and "1520837563454,2018-03-12-14-52-43,INFO,Utilities,"Removing backup file \"C:\\Users\\Administrator\\Documents\\en\\BS_Create_VMware.html\" from index because it does not exist in destination"".
- Click [Close] to quit.
!
It is strongly recommended to perform the Cyclic Redundancy Check (CRC) regularly to ensure the data integrity of the backup data files and clear out the incomplete files from backup destination.